Meta-analysis using HLM 1 Running head: META-ANALYSIS FOR SINGLE-CASE INTERVENTION DESIGNS Comparing Two Meta-Analysis Approaches for Single Subject Design: Hierarchical Linear Model Perspective

In spite of the criticism for lack of validity, researchers still use single-subject designs to evaluate school counseling when working with individual students, autism treatment and many other individualized therapeutics interventions. The accumulation of many studies on one area of research creates the need for comparing the efficacy of the findings from these studies. Researchers developed several meta-analyses techniques to synthesize findings from single subjects’ studies, including a recent application of the HLM to conducting meta-analyses of single-subject designs. Two alternatives for estimating and modeling effect sizes from single subject steadies are presented through a comparison of two approaches that utilize Hierarchical Linear Modeling. Meta-analysis using HLM 3 Introduction Elegant experimental studies characterized by random assignment, control groups, and preand post-intervention descriptive data, some cannot be applied to a research investigate individualized treatments or interventions. On the contrary, research studies involving individualized treatments are more likely to employ samples of convenience from applied settings where a single intervention is examined and withdrawal of treatment is unethical. Finally, such intervention studies have employed single-case methodology, where results are expressed in graphic displays that do not readily lend themselves to traditional meta-analytic investigations (McConnell, 2002). Single-subject designs became a common method for analyzing and evaluating the effectiveness of individualized therapeutic interventions. Although single-case studies are cumbersome for use in meta-analyses, such designs nevertheless have high statistical control and excellent internal validity, and allow for analyses of trends in data across time (Kratochwill, Mott, & Dodson, 1984; McConnell, 2002). Quantitative analysis of multiple single-case studies is particularly valuable in the areas of individual therapeutic interventions, areas that are characterized by diverse populations that challenges study replication (Van den Noortgate & Onghena, 2003; White et al., 1989). Despite the complexity involved, examination of intervention effects across multiple single-case studies using meta-analysis can contribute to research practice. Meta-analysis is a preferred methodology for deriving objective conclusions about the effectiveness of several individual therapeutic interventions across a series of studies (White, Rusch, Kazdin, & Hartmann, 1989). Previously employed methods for Meta-analysis using HLM 4 conducting meta-analyses of single-case studies have included the conversion of singlecase graphic displays to quantitative data for the calculation of effect-sizes (Clark & Stewart, 1997); a piecewise regression technique (Center, Skiba, & Casey, 1985-86); percent of non-overlapping data between baseline and treatment phases (Scruggs, Mastropieri, & Casto, 1987); calculation of effect sizes for changes between phases (Corcoran, 1985; White, Rusch, Kazdin, & Hartmann, 1989); and pooling data from multiple baseline and treatment phases (Busk & Serlin, 1992). The majority of these developments focused on finding appropriate methods for estimating effect sizes for the different methods adopted in single-subject designs (Corcoran, 1985). This paper focuses on comparing two alternatives for estimating effect sizes using Hierarchical Linear Models (HLM), and modeling their variability across several singlecase studies. One alternative estimates and models the effect sizes using the actual repeated observations within the phases of single-case studies in HLM repeated measure designs (Van der Noortgate & Onghena, 2003). The other uses the “V-Known” procedure of HLM described by Raudenbush and Bryk (2002) on calculated effect sizes by Busk and Serlin’s (1992) method for calculating effect sizes from single-case studies, and moderate the variability of the effect sizes by subject or study characteristics. Effect Size in Single-Subject Design Single subject design can be described as a special form of repeated measures designs. Data from such designs often lack the property of independent observations that can lead to the possibility of having autocorrelations across measures. Researchers often put forward assumptions such as normality and sphericity that pertain to the nature of the Meta-analysis using HLM 5 data in the populations of such designs. Such assumptions are often hard to achieve, and, unless they can be validated, misleading findings may result (Myers & Well, 2003). In its simplest form of (AB) design, a single-subject design involves repeatedly observing a subject on some outcome over a period of time during a baseline period (phase A) and during the period of an intervention (phase B). The objective is to have a consistent assessment of the effectiveness of the intervention. There may be an unequal number of observations recorded in phases A and B. Variations of the design can take the form of having multiples of each of the two phases, such as ABAB design, or having an additional phase (C) that represents a different intervention from the one in B, such as ABC, in the same study. Ultimately, observations from the phases are compared either graphically (Busk & Serlin, 1992) or by using some permutation tests (Edgington, 1995) to report on the effectiveness of an intervention. In 1992, Busk and Serlin described three approaches for calculating effect size for measuring treatment effectiveness in single-case study. These approaches vary according to the assumptions needed for such designs. Using the first approach where no assumptions are made, an effect size, similar to Glass’s “d” effect size (1976), is found for each subject by dividing the difference between the means from the two phases of the study by the standard deviation of the observed values in phase A under the null hypothesis of no intervention effect. Effect sizes calculated under this approach have a binomial distribution if they are estimating a common population effect size with some effect sizes being positive and other are negative (Busk & Serlin, 1992). Using the binomial sign-test model, a confidence interval can be established for the population median effect size. Meta-analysis using HLM 6 In the second approach described by Busk and Serlin assuming equal variances and similar distributions for the observations in both phases, an effect size for each subject is found by dividing the difference between the means from the two phases by the square root of the pooled variance from both phases. Pooling variances under the given assumption allows for the attaining a more precise estimate for the denominator of the effect size. The assumption of similar population distributions of the outcome for the two phases with equal variances suggests that the distribution of the effect sizes is symmetrical, a property that can be adopted under the Wilcoxon model to find confidence intervals for the population median of the effect sizes (Busk & Serlin, 1992). For the third approach, effect sizes are obtained the same way as in the second approach under the assumption of normally distributed observations in both phases and the sphericity assumption. The estimated effect sizes have a non-central t-distribution that can be used to build confidence intervals for each study effect size as well as a confidence interval for an overall effect size. For this study we use effect sizes estimated by Busk and Serlin’s third approach in the variance-known (V-known) modular of hierarchical linear models (HLM) (Raudenbush & Bryk, 2002). Variance estimates of the effect sizes are based on Hedges, 1994, approach for correlated observation. Purpose and Method The purpose of this study is to compare two approaches of conducting metaanalysis for single-subject studies via HLM when studying the effectiveness of therapeutic interventions. While the two approaches rely on using HLM for combining data from single-subjects studies, the difference between them is in their treatment of the Meta-analysis using HLM 7 data within HLM. In one approach, the actual observations from single-subject studies are used in a two level HLM analysis. In the second approach, separate effect size based on Busk and Serlin’s (1992) method with its estimated variance for each subject is used in the “V-known” version of HLM. A more detailed description of the two approaches is presented in the next two sections. There are at least three benefits of using HLM in combining data for singlesubject studies. One is obtaining better estimates of treatment effectiveness by “borrowing information” from other similar studies (Radenbush & Bryk 2002). HLM estimates of effect sizes are weighted combination of the individual effect size estimated weight by its precision and the overall effect size from all the studies in the combined data. A second benefit is the accommodation of dependent repeated observations. An important assumption required by many statistical procedures and certainly hard to deny in single-subject studies due to their design nature. Finally, the ability to model the systematic variability in studies’ effect sizes by subjects or studies’ characteristics. Data Data for this study come from a larger data set under current analysis by the author. The full data set consists of 233 participants with autism who were given one of seven different forms of intervention. Three outcomes were observed on two different scales (percentages and frequencies). Participants were identified by the type of treatment received, the outcome measured, and the scale of the data (percent versus frequency). For the purpose of this study, data from 35 participants with frequency data observed on academic/communication outcome. Twenty three subjects were introduced to the “Ecological and Milieu” type of intervention, while the remaining 12 were introduced to Meta-analysis using HLM 8 the “Peer Supported” type of intervention. Data from all 35 participants were combined to form one sample to examine effect size estimate properties from a larger sample size. The two approaches also were compared separately for the two types of interventions, under moderate (23 participants) and small (12 participants) sample sizes. Applying Two Level HLM Model for Combining Single-subject Data Hierarchical linear models are statistical methods for analyzing hierarchically structured data where variables are measured at different levels. For example, measures might be available on patients as well as doctors where patients are nested within doctors. This hierarchy in the data can create a lack of independent observations. The use of the hierarchical linear model accounts for the dependency between the scores on the first level. The basic structure of HLM in its simplest form (two levels HLM) consists of two or more regression models for its two levels. Characteristics measured on level-one units are used in a regression equation to model an outcome of interest within each level-two units. Estimates of the regression coefficients from level-one model are then used as outcomes in one or more regression equations to be modeled by level-two characteristics (variables). Data from single-subject studies fit nicely into the structure of hierarchical data. Repeated observations on the individual from the phases of each study constitute the first level of the HLM model as